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Abstract 

We introduce two different approaches for cluster- 
ing semantically similar words. We accommodate 
ambiguity by allowing a word to belong to several 
clusters. 

Both methods use a graph-theoretic representa- 
tion of words and their paradigmatic relationships. 
The first approach is based on the concept of cur- 
vature and divides the word graph into classes of 
similar words by removing words of low curvature 
which connect several dispersed clusters. 

The second method, instead of clustering the 
nodes, clusters the links in our graph. These contain 
more specific contextual information than nodes 
representing just words. In so doing, we naturally 
accommodate ambiguity by allowing multiple class 
membership. 

Both methods are evaluated on a lexical acquisi- 
tion task, using clustering to add nouns to the Word- 
Net taxonomy. The most effective method is link 
clustering. 

1 Introduction 

Graphs have been widely used to model many 
practical situations (Chartrand , 1985| l, including 
many semantic issues: The link structure of 
the World Wide Wed has been investigated 
and manipulated to detect shared interest com- 
munities ( Eckmann and M oses, 2002| ), and mod- 
eling WordNet as a graph has yielded in- 
sight about semantic relatedness and ambiguity 
( Sigm an and Cecchi, 2002| >. 

In this paper, we present a graph model for nouns 
and paradigmatic relationships collected from the 
British National Corpus (BNC) 1 using simple lex- 
icosyntactic patterns. 



The resulting semantic structure can be used for 
lexical acquisition (by gathering nodes into clusters 
and labeling the clusters) and word sense discrimi- 
nation (by determining when a node in the graph is 
really a conglomeration of nodes representing dif- 
ferent senses). 

We introduce two tools to approach 
these tasks: the curvature measure of 
( |Eckmann and Moses, 2002|) and the Markov 
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Clustering (MCL) of <|yan Dongen, 2000) . The first 
algorithm removes the nodes of low curvature (the 
hubs of the graph), upon which the word graph 
breaks up into disconnected coherent semantic 
clusters. MCL decomposes the word graph into 
small coherent pieces via simulation of random 
walks in the graph which eventually get trapped in 
dense regions, the resulting clusters. 

Both methods effectively place each node into 
exactly one cluster, breaking the graph into 
equivalence classes. The shortcomings of any 
such approach become apparent once we consider 
ambiguity — when each word is treated as an indivis- 
ible unit in the graph, we need to split these seman- 
tic atoms to account for different senses. We inves- 
tigate an alternative approach which treats each in- 
dividual coordination pattern as semantic node, and 
agglomerates these more contextual units into usage 
clusters corresponding closely to word senses. 

A comparative evaluation of these methods on a 
lexical acquisition task is presented in Sect. |5] 

2 The graph model 

To build a graph representing the relationships 
between nouns, we used simple regular expres- 
sions to search the BNC, which is tagged for 
parts of speech, for examples of lexicosyntactic 
patterns which are often indicative of a seman- 
tic relationship (Hear st, 1992| ). The hypothesis 
is that nouns in coordinations are semanti- 




Figure 1 : Local graph around body 



cally similar (cf. Riloff and Shepherd (1997 ), 



Roark and Charniak (1998 ), 

Widdows and Dorow (2002 )). We collected 
coordinations of noun phrases using simple pat- 
terns, dropped prenominal modifiers, and built a 
word graph by 

1. Introducing a node for each of the nouns; 

2. Connecting two nouns by an edge if they co- 
occurred in a coordination. 

Consider the following example sentences drawn 
from the BNC containing a coordination "body" 
appearing in: 

Legend has it that the mandarin was so grate- 
ful to Earl Grey for services rendered that he 
gave him his secret tea recipe, to keep mind, 
body and spirit together in perfect harmony. 

So far as ordinary citizens and non- 
governmental bodies are concerned, the 
background principle of English law is that a 
person or body may do anything which the 
law does not prohibit. 

Christopher was also bitten on the head, neck 
and body before his pet collie Waldo dashed 
to the rescue. 

The highlighted coordinations give rise to edges 

body<^mind, body^spirit, mind^spirit 
body^person 

body^head, body^neck, head^neck 

in the word graph. Fig. [2 displays a subgraph of 
our word graph centered around body and consist- 
ing of the top 17 neighbors of body and the top 8 
neighbors of the neighbors. The word graph has 
this very simple interpretation: Words which are 
directly linked are semantically close. This graph 



consists of 88, 900 nodes (word types) and 551, 745 
edges. We ignore the order in which two words 
co-occur in a coordination, the edges in our graph 
are not given any direction. To reduce noise, we 
keep only those links in the graph which appear 
in a triangle, since the links within a triangle con- 
firm each other's significance. This results in a re- 
duced word graph consisting of 48, 727 nodes and 
505, 412 edges. 

3 Graph curvature and quantifying 
semantic ambiguity 

Our approach to assessing ambiguity is similar to 
the one proposed by Sproat and van Santen (1998 ), 
in that our measure also quantifies ambiguity 
based on the semantic cohesiveness of the target 
word's neighborhood. Words with a very tightly- 
knit neighborhood are assigned smaller ambiguity 
scores than words whose neighborhood is rather 
fuzzy. 

We measure the semantic cohesiveness of a 
word's neighborhood (and as a result ambiguity) as 
the curvature of the word in the graph. Curvature 
is a property of nodes in a graph which quantifies 
the interconnectedness of a node's neighbors. The 
curvature curv(w) of a node it; is defined by: 



curv(?i>) 



#( triangles w participates in) 
#( triangles w could participate in) 



Curvature is the fraction of existing links among a 
node's neighbors out of all possible links between 
neighbors. It assumes values between and 1. A 
value of occurs if there is no link between any of 
the node's neighbors (i.e. the neighbors are max- 
imally disconnected), and a node has a curvature 
of 1 if all its neighbors are linked (i.e. its neigh- 
borhood is maximally connected). Fig. |2] shows 
nodes of low, medium and high curvature respec- 
tively. Curvature measures whether neighbors of 
a word are neighbors of each other. Very specific 
unambiguous words have high curvature, because 
they usually live in small, semantically very co- 
hesive communities in which many pairs of nodes 
have mutual neighbors. These communities thus 
contain a high density of triangles. Examples for 
tight word communities are the days of the week, 
the world religions, Greek gods, chemical elements, 
English counties, the planets, the members of a rock 
band, etc. Ambiguous words, on the other hand, are 
linked to members of different communities (corre- 
sponding to the different meanings of w) which do 
not know each other. An ambiguous word's neigh- 
borhood thus has a low density of triangles which 
results in a low curvature value. 



Figure 2: Left: curv(w ) = 0, Right: curv(w ) = 0.3, 
Bottom: curv(i> ) = 1. 
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Figure 3: Curvature vs. model frequency 



In information theory, it is common to use the 
negative logarithm of relative word frequency to 
measure a word's information content (info(w) = 
— log(rf(«;))). The intuition is that very frequent 
words tend to be very general and uninformative, 
and that very infrequent words tend to be more spe- 
cific. Among the most frequent words in our model 
are countries, which according to info(-) would be 
wrongly categorized as very uninformative, am- 
biguous words. 

Fig. |3]is a plot of curvature against frequency in 
our model. The countries among the nodes are in- 
dicated by black stars. Very clearly, the curvatures 
of the countries are considerably higher than the av- 
erage curvature of words with similar frequency in 
the model, suggesting that, despite their high fre- 
quency, they are all very informative, i.e. unam- 
biguous. The outlier in the lower left corner of the 
plot is monaco which may not seem ambiguous, but 
which has several different meanings in the BNC: 
country, city, 14th century painter and 20th cen- 
tury tenor (cf. Fig. 0J). To check how well cur- 



Figure 4: Local graph around monaco 



Table 1 : Rank correlations between any two out of 
number of WordNet senses, word frequency in the 
model, degree and curvature. 





senses freq 


deg 


curv 


senses 


1.000 0.475 


0.480 


-0.538 


freq 


1.000 


0.963 


-0.865 


deg 




1.000 


-0.884 


curv 






1.000 



vature is suited for detecting and assessing ambigu- 
ity, we take all words in our model which are listed 
in WordNet and check how strongly curvature and 
the number of WordNet senses are related. Since 
the relationship does not have to be linear, we re- 
place curvature and number of WordNet senses by 
their ranks before computing the Pearson correla- 
tion coefficient. We also wanted to see whether and 
to which degree curvature better reflects ambigu- 
ity than a word's frequency in the model or its de- 
gree (the number of links attached to a node) in the 
graph. Tablefjlists the mutual Pearson correlations 
between any two quantities out of model frequency, 
degree, curvature and number of WordNet senses. 
Our analysis shows that with a negative correlation 
of —0.538, curvature is more strongly related to the 
number of WordNet senses and thus a better mea- 
sure of ambiguity than model frequency or degree. 

4 Inducing classes of similar words 

A semantic category (also referred to as a semantic 
field) is a grouping of vocabulary within a lan- 
guage, organizing words which are interrelated 
and define each other in various ways. The acqui- 
sition of semantic categories from text has been 
addressed in several different ways: Work in this di- 
rection can be found in (Perei ra et al. (1 993 ), 



Schutze (19951, |Pantel and Lin (2002) , 

Dorow and Widdows (2001) ). 

Word clustering techniques differ in the way they 
assign words to clusters, either allowing words to 
belong to several clusters (soft clustering), or as- 
signing words to one and only one cluster (hard 
clustering). A problem of hard clustering tech- 
niques is that each word is coerced into a single 
cluster irrespective of whether it is closely associ- 
ated with other clusters, too. 

Semantic categories overlap considerably, but 
hard clustering produces mutually exclusive clus- 
ters and forces ambiguous words to associate with 
a single cluster only. We therefore concentrate on 
soft clustering. 

4.1 Graph clustering 

In the following, we describe two approaches to soft 
clustering of words in our graph. 

Curvature clustering: In our word graph, am- 
biguous words function as bridges between differ- 
ent word communities, e.g. cancer is the meeting 
point of the animal community, the set of lethal 
diseases and the signs of the zodiac. By remov- 
ing these "semantic hubs", the graph decomposes 
into small pieces corresponding to cohesive seman- 
tic categories. In detail, the method for extracting 
clusters of similar words is the following: 

1. Compute the curvature of each node in the graph. 

2. Remove all nodes whose curvature falls below a 
certain threshold (0.5). 

3. The resulting connected components constitute 
clusters of semantically similar words. 

Application of this algorithm to our word graph 
results in 700 clusters of size > 2. The resulting 
clustering covers 2, 306 of the nouns in our model 
with 21, 218 of the nodes not making the 0.5 curva- 
ture threshold and 25, 203 isolated nodes. 

This method produces a hard clustering of the 
high curvature words. Since high curvature words 
have a well-defined meaning, we expect a hard clus- 
tering approach to detect the (unique) semantic cat- 
egory each of these words belongs to. 

Curvature clustering in this form cannot give in- 
formation on the semantically fuzzy low curva- 
ture words. Therefore, we augment each of the 
clusters with the nodes attached to it. Table |2] 
lists some of the enriched clusters. The origi- 
nal cluster (the core of the extended cluster) is 
printed in bold font, cluster neighbors which did 
not pass the curvature threshold are highlighted in 
italics, and neighbors which were isolated in the 
initial clustering are printed in normal font. Of- 
ten, the core words of high curvature are quite spe- 



Table 2: Clusters resulting from the curvature ap- 
proach 



applewood fruitwood cherry ivory pine oak 



jainism sikhism vaisnavism islam buddhism hinduism Christianity 
judaism 



horseflies lacewings butterfly mosquito beetle centipedes ladybird 
bird moth 



freestyle backstroke butterfly race medley 



printmaker ceramicist sculptor painter draughtsman artist 



pomelo papaya banana potato pineapple mango peach palm pear 
parsnip 



poliomyelitis tetanus tb kinase cough polio diphtheria malaria 
disease tuberculosis pertussis anthrax 



thiamin niacin riboflavin fibre protein iron calcium 



oratorio cantata concert baroque opera aria motet play 



morphine methadone chloroform heroin caffeine length phos- 
phate cocaine Isd librium metabolite 



stepsister stepbrother friend father sister stepmother brother 



insectivores artiodactyls ungulates mammal herbivore individual 
carnivore rodent horse order fruit 



cosine tangent area sine torsion factor 



cific and unambiguous, suggesting that high curva- 
ture is a desirable property for 'seed words' (as in 
( |Roark and Charniak, 19 98 )) used for this purpose. 
By extending the core clusters to their neighbors, 
coverage could be increased to 7, 532 nodes in the 
graph. 

Markov Clustering: A very intuitive 
graph clustering algorithm is Markov Clus- 
tering (http://micans.org/mcl/) developed by 
van Dongen (2000). Markov Clustering (MCL) 
partitions a graph via simulation of random walks. 
The idea is that random walks on a graph are 
likely to get stuck within dense subgraphs rather 
than shuttle between dense subgraphs via sparse 
connections. 

MCL computes a hard clustering. The nodes in 
the graph are divided into non-overlapping clusters. 
Thus, nodes between dense regions will appear in 
a single cluster only, although they are attracted 
by different communities. Inspired by Schutze's 
method (Schutze, 1998) we next replace clustering 
of word strings by clustering of word contexts. 

4.2 Clustering the link graph 

We consider pairs of words which we linked ear- 
lier, as word contexts. For example, organ occurs in 
contexts {organ, piano), {organ, harpsichord), {or- 
gan, tissue) and {organ, muscle). In contrast to the 
semantic "fuzziness" of organ, each of its contexts 
has a sharp-cut meaning and refers to exactly one of 
the senses of organ. By clustering word contexts 
as opposed to clustering the words themselves, a 
word's different meanings can be distributed across 
different clusters which are then interpreted as word 



senses. E.g. we can assign (organ, piano) and (or- 
gan, harpsichord) to one context cluster, and (or- 
gan, tissue) and (organ, muscle) to another different 
context cluster. 

In the setting of Sect. 13 words correspond to 
nodes in the word graph and word contexts coin- 
cide with the graph's edges (with each edge being a 
context of the two nodes it joins). We now consider 
edges as the fundamental nodes of the link graph G' , 
and define the edges of G' as follows: We construct 
the word graph's associated link graph, G', by (see 
Fig. 

1 . Introducing a node ni for each link I in the origi- 
nal graph G. 

2. Connecting any two nodes and ni 2 in G' if l\ 
and I2 co-occurred in a triangle in G. 

The two component words u and v of a context 
/ = (u, v) disambiguate each other, e.g. in the (or- 
gan, harpsichord) context, both organ and harpsi- 
chord are instruments, since this is the intersection 
of all the possible meanings of organ and all the 
possible meanings of harpsichord. The nodes ni in- 
troduced in step 1 therefore have a much narrower 
meaning than the nodes in G. 

The links of a triangle in G constitute mutually 
overlapping word contexts. We therefore expect the 
links in such a context triangle to have the same 
"topic", and the nodes at the corners of the trian- 
gle to have the same meaning. This means, step 2 
connects two nodes and ri/ 2 if the corresponding 
contexts l\ and I2 are semantically similar. 

Fig. |5]shows the local word graph around organ. 
Its associated link graph is illustrated in Fig.|6l(only 
those connected components containing organ with 
more than one node are displayed). Note that in 
the link graph, neighbors corresponding to different 
senses of organ are no longer linked. 

Instead of clustering words by partitioning the 
original graph G, we cluster word contexts by par- 
titioning G"s associated link graph G' . The nodes 
n\ in G' are built with contextual information, and 
thus typically have a clear-cut meaning. With little 
(if any) ambiguity left in the link graph, a hard clus- 
tering algorithm, such as MCL, is fit for dividing the 
contexts into (non-overlapping) similarity classes. 
In detail, our algorithm is: 

1) Start with the original graph G. 

2) Construct the associated link graph G'. 

3) Apply Markov Clustering to G' . 

4) Merge clusters whose overlap in information ex- 
ceeds a certain threshold. 

The clustering resulting from step 3 is too fine- 
grained. Several of the context clusters describe the 
same "topic". We collapse these multiple clusters 




Figure 5: Local word graph around organ 




Figure 6: Its associated link graph 



via another application of MCL, this time applied to 
a graph of context clusters which are linked if their 
shared information content (the negative logarithm 
of the probability of the words they have in com- 
mon) exceeds 50% of the information contained in 
the smaller of the two clusters. Step 4 reduced the 
12,786 clusters resulting from step 3 to a total of 
5, 849 clusters. 

5 Comparative evaluation for lexical 
acquisition 

One of the principal uses of word clustering 
techniques is to supply missing lexical informa- 
tion. For example, the hypernyms of a word a 




can often be inferred from the hypernyms of its 
neighbors bi,...,b n . This property was used by 
(Hearst and Schiitze, 1993 ) and ( Widdows, 2003 ) to 
map unknown words into the WordNet taxonomy. 
The accuracy of such methods depends on the tax- 
onomy in question, the method used to obtain the 
neighbors b\, . . . , b n , and the specificity of the re- 
sult desired. 

From the subset of nouns in our test graph known 
to WordNet, we randomly picked a set of 1 ,200 test 
words consisting of 600 proper nouns and 600 com- 
mon nouns. Each of these two subsets is further di- 
vided into 3 frequency categories (top (500-1000), 
mid (250-1000), low (below 250)) which consist of 
200 words each. 

Pretending that we don't know the test words, 
we test how well we do in re-mapping them into 
WordNet. For each of the test words t, we look up 
which clusters it appears in and keep its most simi- 
lar cluster c max . Similarity between words and clus- 
ters is computed using cosine similarity between 
their vector representations in a vector space model 
dDeerwester et al, 19901 . 

We then assign a sense label to c max us- 
ing the sense-labeling algorithm proposed in 
(Wid dows, 200 3 ) which treats any hypernym of any 
of the cluster members as a potential cluster label. 
Potential cluster labels are rated based on two com- 
peting criteria: 

• The more cluster members a label subsumes, the 
better (favoring more general labels). 

• The more informative the label, the better (favor- 
ing more specific labels). 

Since including the test word t in the sense-labeling 
process would be using information about t which 
we are not given in a real lexical acquisition situa- 
tion, we disregard both, the test word and the cluster 
members which are morphologically related to the 
test word. 

The labeling algorithm outputs a cluster's top five 
labels together with a score assessing their ade- 
quacy. We compare each of these labels with each 
of the test word's ancestors in WordNet, and, in case 
of a match, record the number of intervening levels 
between the test word and the label. E.g. the test 
word opera appears in the cluster (jazz, music, fes- 
tival, sound, beat, reggae, soul, ballet, funk, coun- 
try, orchestra, film, table, poetry). Table |3] shows 
the cluster labels and scores assigned by the class- 
labeling algorithm. Column match lists the number 
of intervening WordNet levels between opera and 
each of the labels. 

If a test word t is not covered by the clustering, 
we do a depth-first search on the original word 



graph stalling at t and moving along the strongest 
link until we reach a node t' covered by the cluster- 
ing. We then pretend that t belongs to the cluster(s) 
t' appears in. To summarize, evaluation consists of 



Table 3: Labels assigned to the opera cluster 



label 


score 


match 


auditory communication 


0.438 


4 


communication 


0.076 


5 


abstraction 


-0.265 


8 


relation 


-0.500 


7 


social relation 


-0.603 


6 



the following steps. For each test word t, 

1. If t doesn't appear in any cluster, follow strongest 
links until you reach a word t' which is covered by 
the clustering and substitute t with t'. 

2. Collect the clusters t appears in. 

3. Compute the similarity between t and each of 
the clusters and keep only the cluster c max which is 
most similar to t. 

4. Compute a class label I for c max \ {t}. 

5. Check if (and how closely) I corresponds to one 
of t's WordNet senses. 

Basis for comparison We use the following sim- 
ple sense-labeling method as basis for comparison. 
For each test word t, we find its nearest neighbor 
n in the graph. For all the hypernyms of t and 
n, we find their common subsumer cs(t, n) which 
minimizes the average distance to t and n. We are 
directly using taxonomic knowledge about our test 
word t to find the optimal position in the WordNet 
tree where t and n should join. In a real lexical ac- 
quisition situation, of course, such information is 
not available. This method therefore forms a sim- 
plest upper bound on how well we could expect to 
do in mapping unknown words into the WordNet 
taxonomy. 

Results Table |4] summarizes the performance of 
the algorithms on the lexical acquisition task de- 
scribed above (similar results are obtained for 
proper nouns). For each test set and each method, 
row N lists the number (percentage) of test words 
which are not in WordNet. The number (percent- 
age) of words which received a label not corre- 
sponding to any of its senses with any number of 
intervening WordNet levels is listed in row W. The 
rows i = 1..12 contain the number of words which 
were assigned a correct label with i or less interven- 
ing WordNet levels. For these rows, the percent- 
ages in parentheses are relative to the total number 
of words which were assigned a correct label. 



Naturally, the baseline method, which is using 
(normally unknown) taxonomic information about 
the test word itself, performs best. Of all the meth- 
ods, it has the fewest number of wrongly assigned 
labels (row W). An accuracy of 92%, resp. 99% is 
reached at < 5 intervening WordNet levels. 

6 Conclusions 

Among the other three methods, Markov Cluster- 
ing (MCL) of the link graph outperforms both MCL 
on the original graph and curvature clustering. The 
number of wrongly assigned labels is about half of 
those for curv and orig and the values in the 12 
rows are consistently higher with an accuracy of 
over 85% at < 6 WordNet levels. The link graph 
clustering therefore produces more accurate labels. 
In the top frequency category, MCL on the original 
graph has a slightly higher percentage of correctly 
assigned class labels for small numbers of interven- 
ing WordNet levels, but is soon overtaken by the 
link graph clustering. The lower values for the cur- 
vature clustering can be partly explained by its low 
coverage. 854 of the 1, 200 test words were not cov- 
ered by the curvature clustering and had to be traced 
to clusters using depth-first search in 1 to 46 steps 
(with 60% (80%) of the test words being at most 3 
(6) links apart from a cluster). 

Judging by the classes in Table we expect cur- 
vature clustering to do especially well in recogniz- 
ing the meanings of words unknown to WordNet. 
We have shown that graphs can be learned directly 
from free text and used for ambiguity recognition 
and lexical acquisition. We introduced two new 
combinatoric techniques, graph curvature and link 
clustering, and evaluated their contribution as clus- 
tering methods for lexical acquisition. Link clus- 
tering produces particularly promising results when 
compared with information in the WordNet noun hi- 
erarchy. These results demonstrate that our com- 
binatoric methods for analysing the geometry and 
topology of graphs improve language learning. 
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Table 4: Evaluation results for common nouns 



Not 

Wrong 

1 

2 

3 

4 

5 

6 

7 

8 

9 

[0 

] I 

12 



Not 

Wrong 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

1 1 

12 



Not 

Wrong 

1 

2 

3 

4 

5 

6 

7 

8 

9 
10 
11 
12 



2(0.01) 

74(0.37) 

21(0.17) 

40(0.32) 

66(0.53) 

90(0.73) 

99(0.80) 

108(0.87) 

1 14(0.92) 

118(0.95) 

121(0.98) 

121(0.98) 

122(0.98) 

123(0.99) 



nnljop 



2(0.01) 

32(0.16) 

23(0.14) 

47(0.28) 

79(0.48) 

106(0.64) 

128(0.77) 

143(0.86) 

153(0.92) 

161(0.97) 

164(0.99) 

164(0.99) 

166(1.00) 

166(1.00) 



2(0.01) 

69(0.34) 

8(0.06) 

26(0.20) 

50(0.39) 

70(0.54) 

89(0.69) 

103(0.80) 

110(0.85) 

120(0.93) 

124(0.96) 

127(0.98) 

129(1.00) 

129(1.00) 



2(0.01) 

14(0.07) 

38(0.21) 

62(0.34) 

84(0.46) 

99(0.54) 

183(0.99) 

183(0.99) 

183(0.99) 

184(1.00) 

184(1.00) 

184(1.00) 

184(1.00) 

184(1.00) 



0(0.00) 

65(0.33) 

19(0.14) 

36(0.27) 

63(0.47) 

86(0.64) 

100(0.75) 

111(0.83) 

124(0.93) 

128(0.96) 

130(0.97) 

132(0.99) 

134(1.00) 

134(1.00) 



0(0.00) 

38(0.19) 

17(0.11) 

45(0.28) 

74(0.46) 

105(0.65) 

127(0.79) 

139(0.86) 

150(0.93) 

156(0.97) 

158(0.98) 

159(0.99) 

161(1.00) 

161(1.00) 



0(0.00) 

77(0.39) 

9(0.07) 

22(0.18) 

43(0.35) 

60(0.49) 
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